Multi-channel data sonification employing data-modulated sound timbre classes

ABSTRACT

Multichannel data sonification arrangements for providing parallel perceptual channels for representing complex numerical data to a user seeking to identify correlations within the data are presented. In an implementation, several varying data quantities are represented by time-varying sound and presented to a user to observe both variations and correlations among sonic events or trends. The data sonification can include multiple data-modulated sound timbre classes, for example each separately rendered within a stereo sound field. The sound timbre classes can be arranged to convey a sonic metaphor.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/176,000, filed Feb. 7, 2014, which is a continuation of U.S.application Ser. No. 13/846,843, filed Mar. 18, 2013, now U.S. Pat. No.8,692,100, issued Apr. 8, 2014, which is a continuation of U.S.application Ser. No. 13/450,350, filed Apr. 18, 2012, now U.S. Pat. No.8,440,902, issued May 14, 2013, which is a continuation of U.S.application Ser. No. 12/817,196, filed Jun. 17, 2010, now U.S. Pat. No.8,247,677, issued Aug. 21, 2012, the disclosures of each of which areincorporated herein in their entireties by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention generally relates to data sonification and inparticular to sound rendering allowing for multiple simultaneouschannels of information-carrying utilizing at least the timbre of one ormore parameterized audio waveforms.

Background of the Invention

Sonification is the use of non-speech audio to convey information orperceptualize data. Due to the specifics of auditory perception, such astemporal and pressure resolution, sonification offers an interestingalternative or complement to visualization techniques, gainingimportance in various disciplines. Sonification has been wellestablished for a long time already as Auditory Display in situationsthat require a constant awareness of some information (e.g. vital bodyfunctions during an operation).

Many analytic tool outcomes produce data that lend themselves well tohelpful visualizations (in geographic, spatial formats, and abstractformats). In highly cluttered visual displays, advanced datasonification can be used to convey yet additional data without furtherencumbering the visual field.

However, sonification systems have long remained far too primitive orinappropriate for general data sets, visualization environments, GISapplications, etc. Accordingly, despite much interest and ongoingintuitive promise, data sonification has remained a novelty area and theuse of sonification as a method for exploration of data and scientificmodeling is an ongoing topic of low-level research.

Nonetheless, work has demonstrated that sonification can be an extremelypowerful tool is if data is expressed in terms of parameterized timbrevariations coupled with systematic sonic design. With proper sonicdesign (not unlike proper visual design) rich powerful multichannel datarepresentations are possible wherein several channels of data values canbe simultaneously conveyed effectively.

So empowered, data sonification takes on the same types of support needsand multi-parameter handling that would be afforded sophisticated datavisualization systems. As a result, data sonification can take a peerrole with data visualization and accordingly the two can share many ifnot all of the same data preprocessing operations and environments.

Thus the present invention is directed to parameterized timbrevariations, audio signal and sonic design, broader sonificationenvironments, interactions with visualization environments, and otherrelated aspects important to making data sonification the viable andpowerful tool it could be.

SUMMARY OF THE INVENTION

The invention integrates data sonification tools to provide practical,useful sonification representations for data that would otherwiseclutter visually busy or crowded graphical GIS displays. Although alwaysseeming to hold interesting promise, sonification to date is often notvery useful or practical. The invention provides for use of a family ofsignal synthesis, control, and metaphor techniques and technologies forexamining environmental, science, business and engineering datasets.

The invention comprises “multi-channel sonification” usingdata-modulated sound timbre classes set in a spatial metaphor stereosound field. The sound field can be rendered by inexpensive 2D speakerand 2D/3D headphone audio, resulting in an arrangement providing aricher spatial-metaphor sonification environment.

The invention includes deeply integrated data sonification of data fromvarious models, tools, and data sets, and provides sonification ofanalysis output data as well as selected measurement data. In oneapplication of the invention, sonification will be rendered either in aGIS display context or in an abstract data set context, as appropriate.Further, sonification can be used to support interactive adjustments ofanalysis tools.

The user can navigate a listening point within a data sonification.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of the presentinvention will become more apparent upon consideration of the followingdescription of preferred embodiments, taken in conjunction with theaccompanying drawing figures.

FIG. 1 depicts a comparison between representing numerical data viavisualization and sonification.

FIG. 2a depicts a representative temporal assessment equivalence betweenlooking at a static graphic and listening to a static sound field.

FIG. 2b depicts a representative temporal assessment equivalence betweenlooking at a time-varying graphic and listening to a time-varying soundfield.

FIG. 3 depicts then how data visualization and data sonification canprovide parallel channels in representing complex data resident in acomputer to a human attempting to comprehend it.

FIG. 4a provides a representational view of four example issues thatlead to the need for careful sonic design so as to effectively carrymultiple channels of information simultaneously.

FIG. 4b depicts how temporal variation of timbre, pitch, and amplitudeattributes at rates notably less than 50 msec/20 Hz are perceived as achange in these attributes, while temporal variation of timbre, pitch,and amplitude attributes at rates notably more than 50 msec/20 Hz areperceived as quality of timbre of the tone.

FIG. 5 depicts how the parallel channels of data visualization and datasonification can be used in representing complex numerical data residentin a computer to a human attempting to find correlations within thecomplex numerical data.

FIG. 6 depicts exemplary generation of a pulse waveform from a thresholdcomparison of an adjustable threshold value with the amplitude of aperiodic ascending ramp waveform.

FIG. 7 depicts exemplary generation of a pulse waveform from a thresholdcomparison of an adjustable threshold value with the amplitude of aperiodic descending ramp waveform.

FIG. 8 depicts exemplary generation of a pulse waveform from a thresholdcomparison of an adjustable threshold value with the amplitude of aperiodic triangle waveform.

FIG. 9 depicts exemplary generation of a pulse waveform from a thresholdcomparison of an adjustable threshold value with the amplitude of aperiodic sinusoidal waveform.

FIG. 10 shows “multichannel sonification” using data-modulated soundtimbre classes set in a spatial metaphor stereo sound field.

FIG. 11 shows an exemplary embodiment where dataset is provided tosonification mappings controlled by interactive user interface.

FIG. 12 shows an exemplary embodiment of a three-dimensional partitionedtimbre space, allowing the user to sufficiently distinguish separatechannels of simultaneously produced sounds, even if the sounds timemodulate somewhat within the partition.

FIGS. 13a-13c shows trajectories through a three dimensional timbrespace.

FIG. 14 shows an example of how, through proper sonic design, eachtimbre space coordinate may support a larger plurality of partitionboundaries.

FIG. 15 depicts an exemplary approach for mapping a data value lyingwithin a pre-defined range to a value within a pre-defined range for aparameterized data or cell presentation attribute.

FIG. 16 depicts an exemplary arrangement and general organization ofexemplary pre-visualization operations wherein a native data set ispresented to normalization, shifting, (nonlinear) warping, and/or otherfunctions, index functions, and sorting functions.

FIG. 17 shows an exemplary arrangement wherein interactive user controlsand/or other parameters are used to assign an index to a data set andwherein a selected metaphor is used to automatically generate parameterassignments and graphics rendering operations.

FIG. 18 depicts an exemplary topological interconnection of data flowpaths linking various elements.

FIG. 19 depicts an exemplary adaptation of the arrangement depicted inFIG. 18 configured to selectively direct individually parameters to berendered within a visualization, within a sonification, or within bothsimultaneously

FIG. 20 depicts an exemplary data visualization rendering provided by aGIS system providing an interactive user interface that can be used tooperate a data sonification.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawing figures which form a part hereof, and which show byway of illustration specific embodiments of the invention. It is to beunderstood by those of ordinary skill in this technological field thatother embodiments can be utilized, and structural, electrical, as wellas procedural changes can be made without departing from the scope ofthe present invention. Wherever possible, the same element referencenumbers will be used throughout the drawings to refer to the same orsimilar parts.

Computer-generated data visualization has been actively used to studycomplex data for decades. Data visualization uses parameterize visualprimitives, spatial geometry, time, and other elements to conveynumerical or logical data to a user. In the mid 1980's datavisualization, particularly as used in scientific computing, becameextremely sophisticated. Remnants of this are manifest in medicalimaging system, instrument control consoles, Geographic InformationSystems (GIS), and more recently in newly emergent applications forrepresenting business data (for example employing BIRT).

Data sonification has an interesting but not so successful history. Asdescribed earlier, the present invention reopens the possibilities fordata sonification with controlled sound rendering approaches allowingfor multiple simultaneous channels of information-carrying utilizing atleast the timbre of one or more parameterized audio waveforms.

Before delving into the details of the latter, attention is firstdirected to a framework construction that sets aside many historicnotions of and approaches to data sonification (for example, using pianotones of various pitches to sound out stock and bond market data,computer-event alerts, etc.) and instead treats data sonification as apeer to data visualization.

Comparing Data Visualization and Data Sonification

FIG. 1 depicts a comparison between representing numerical data viavisualization and sonification. All that assumed here is thatvisualization and sonification both are able to carry representations ofnumerical data within them. In the case of visualization, a visualrepresentation of numerical data can take the form of a static graphicor take the form of a graphic that varies in time (i.e., an animation,or an interactive graphic arrangement). Similarly, in the case ofsonification, an auditory representation of numerical data can take theform of a static sound or take the form of a sound that varies in time(i.e., a winking sound, or interactive sound arrangement as with dialinga touchtone phone or playing a music instrument).

In listening to a sound that carries information, time must be taken toascertain the sound's characteristics and what they are configured toconvey. However, this is equally true for looking at graphic thatcarries information: hereto time must be taken to ascertain thegraphic's characteristics and what they are configured to convey. FIG. 2depicts this temporal assessment equivalence between looking at a staticgraphic and listening to a static sound, or more generally, a soundfield as may be created by stereo speakers.

Similarly, FIG. 2b depicts a representative temporal assessmentequivalence between looking at a time-varying graphic and listening to atime-varying sound field. Hereto, both experiences require time toascertain characteristics and what they are configured to convey. Inthese time-varying cases, the time-varying characteristic can be atleast for a time repetitive—implying a symbolic meaning or having arepetitive structure with discernable attributes that can beparameterized—or can vary with time according to underlying timescalesinherent within the numerical data, artificial timescales assignedwithin the numerical data—or can vary with time according user operationof a user interface in an interactive data exploration, simulation, etc.

FIG. 3 depicts then how data visualization and data sonification canprovide parallel channels in representing complex numerical dataresident in a computer to a human attempting to comprehend that complexnumerical data. Of importance in the figure are the visual channel andsonic channel as these have information-carrying capacities, ranges,imposed distortions, and other limitations. These limitations arewell-known for visual representations of data—orange text on bluebackground is hard to see, sensitivity to blue is far less thansensitivity to red, images can get overly crowded, too many thingschanging at once can be hard to follow, changes can only happen so fastor so slow without becoming imperceptible, etc. There are similar typesof limitations for auditory representations of data. Some of these arewell-established and widely agreed upon, such as the lower and higherfrequency range of pitch perception, masking effects, phantomfundamental pitches, minimum event separate time before separate sonicevents are perceived as a single event, beat-frequency effects if twotones are nearly yet not exactly “in tune” with one another, etc. Otherlimitations are known in more specialized settings, for example thesonic designs used in popular music recordings to avoid “clutter” and“muddiness,” and yet other limitations that are not yet well-establishedor necessarily widely agreed upon have come to appear important in thecreation and design of the present invention.

FIG. 4a provides a representational view of four example issues thatlead to the need for careful sonic design so as to effectively carrymultiple channels of information simultaneously. This view is notintended to be precise and in fact if portrayed properly would require ahyper-surface in a four-dimensional space. The salient points are:

-   -   In situations with a plurality of tones are simultaneously        sounding, improper choice of tone type and frequency (or ranges        over which frequencies can vary) can create harmonic        blending/masking effects. Fewer simultaneously sounding tones        can be used to simplify this dimension of concern, but this        limits the information carrying capability of the sound or sound        field.    -   In situations where the underlying data rapidly changes, or        where the rate of a periodic modulation is itself used as a        vehicle for carrying information, increasing the rate of        parameter change “thickens” the tone, which in the limit can        increase sonic clutter (as well as serving as an element of        contrast)    -   As more tones with bigger footprints are sounded simultaneously,        spectral crowding begins to occur making it more difficult to        discern attributes of any one particular sound.    -   As the density of perceived events (wide-swings, rapid change,        etc. in tone attributes) increases, cognitive loading        limitations begin to create confusion.

More specifically as to the last item, FIG. 4b depicts how temporalvariation of timbre, pitch, and amplitude attributes at rates notablyless than 50 msec/20 Hz are perceived as a change in these attributes,while temporal variation of timbre, pitch, and amplitude attributes atrates notably more than 50 msec/20 Hz are perceived as quality of timbreof the tone.

There are other examples, some equally important as those cited here.However non-ideal and intermingled these may seem, there are comparablelimitations in the visual channel. However, well-defined rules andmetrics have been devised long ago to avoid visual channel overloading,distortion, etc., and these well-defined rules and metrics are commonlyaccepted in the practice of fine-art, graphic arts, photography, andother fields including data visualization. In a similar manner,well-defined rules and metrics can be assembled, devised, and refined tofacilitate useful multi-channel data sonification. Several aspects ofthe present invention relate to this in one way or another as will beseen.

Although data sonification can be used by itself, data sonification canalso be used in conjunction with data visualization. Referring again toFIG. 3, the two parallel channels in fact over a number of possibleuses, including:

-   -   Using sonification to offload information carrying capacity from        the visual channel to the sonic channel. Such an approach can be        used, for example, in Geographic Information Systems (GIS) where        the visual channel is typically quite crowded. This can be        useful, for example, when adding additional data presentation        loads to a GIS system, as may be useful for environmental study        and monitoring, etc.    -   Using data sonification and data visualization to reinforce each        other by providing affirming redundancy    -   Using data sonification and data visualization to search for        correlations in complex data—for example several varying        quantities can be portrayed in time-varying graphics and several        other quantities can be portrayed in time-varying sound, an        correlations between sonic and visual events or trends can be        identified—in many cases with a moderately large number of        varying quantities, searching for correlation only with a visual        representation or only within a sonic representation would be        considerable more difficult.

In accordance with the above and other possibilities and opportunities,FIG. 5 depicts how the parallel channels of data visualization and datasonification can be used in representing complex numerical data residentin a computer to a human attempting to find correlations within thecomplex numerical data.

Information Carrying Vehicles for Data Sonification

Attention is now directed toward modulating parameterized periodicwaveforms with low bandwidth signals (most of the spectral energy belowthe lowest pitch that can be heard). First to be considered is pulsewidth modulation, and in particular the creating of pulse widthmodulated waveforms from various source periodic waveforms.

FIG. 6 depicts generation of a pulse waveform generated from a thresholdcomparison of an adjustable threshold value with the amplitude of aperiodic ascending ramp waveform (often referred to as a “sawtooth” or“saw” waveform). A technique using a right-anchored periodic pulse ofcontrollable width, PulseR_(c)(t) is used. Again the waveform may be anelectrical quantity, non-electrical media quantity, or quantityassociated with higher-level signal attributes. Here the periodicup-going ramp waveform typically exhibits a linearly increase from avalue of zero to a value of R 602.1. The reference signal at aparticular instant may be set at a value equal to a proportion Rc 602.2of this, 0≤c≤1. Presenting these waveforms to an appropriate comparatorimplementation whose output values are A_(max)=0 and A_(max)=A 602.3results in the depicted pulse, here PulseR_(c)(t) having value of 0 forthe first 100c % of each period, and the value of A for the remaining100(1−c) % of each period. As the reference signal Rc is raised (capproaches 1 or Rc approaches R), the region of the pulse wave with theamplitude of 0 gets wider, and the region of the pulse wave with theamplitude A_(max) gets narrower. Similarly, as the reference signal Rcis lowered (c approaches 0), the region of the pulse wave with theamplitude of 0 gets narrower, and the region of the pulse with theamplitude A_(max) gets wider.

FIG. 7 depicts generation of a pulse waveform generated from a thresholdcomparison of an adjustable threshold value with the amplitude of aperiodic descending ramp waveform. A technique using a left-anchoredperiodic pulse of controllable width, PulseL_(c)(t) is used. Again thewaveform may be an electrical quantity, non-electrical media quantity,or quantity associated with higher-level signal attributes. Here theperiodic down-going ramp waveform typically exhibits a linearly decreasefrom a value of R 702.1 to a value of zero. The reference signal at aparticular instant may be set at a value equal to a proportion Rc 702.2of this, 0≤c≤1. Presenting these waveforms to an appropriate comparatorimplementation whose output values are A_(max)=A 702.3 and A_(min)=0results in the depicted pulse, here PulseL_(c)(t) having value of A forthe first 100 c % of each period, and the value of 0 for the remaining100(1−c) % of each period. As the reference signal Rc is raised (capproaches 1 or Rc approaches R), the region of the pulse wave with theamplitude of A_(max) gets narrower, and the region of the pulse wavewith the amplitude 0 gets wider. Similarly, as the reference signal Rcis lowered (c approaches 0), the region of the pulse wave with theamplitude of A_(max) gets wider, and the region of the pulse with theamplitude 0 gets narrower.

FIG. 8 depicts generation of a pulse waveform generated from a thresholdcomparison of an adjustable threshold value with the amplitude of aperiodic triangle waveform. Again the waveform may be an electricalquantity, non-electrical media quantity, or quantity associated withhigher-level signal attributes. Here the periodic triangle ramp waveformtypically exhibits a linearly increase from a value of 0 to a value of R802.1 then decrease from a value of R 802.1 to a value of zero. Thereference signal at a particular instant may be set at a value equal toa proportion Rc 802.2 of this, 0≤c≤1. Presenting these waveforms to anappropriate comparator implementation whose output values are A_(max)=A802.3 and A_(min)=0 results in the depicted pulse, here PulseL_(c)(t)having value of 0 for the first 100T(1-c)/2% of each period, the valueof A_(max) from T(1−c)/2 to T(1+c)/2, and the value of 0 for theremaining 100T(1−c)/2% of each period. As the reference signal Rc israised (c approaches 1 or Rc approaches R), the region of the pulse wavewith the amplitude of A_(max) gets narrower, and the regions of thepulse wave with the amplitude 0 on both ends get wider. Similarly, asthe reference signal Rc is lowered (c approaches 0), the region of thepulse wave with the amplitude of A_(max) gets wider, and the regions ofthe pulse with the amplitude 0 on both ends get narrower.

As taught in U.S. patent application Ser. No. 12/144,480 entitled“Variable Pulse-Width Modulation with Zero Constant DC Component in EachPeriod,” the pulse-modulated waveforms of FIGS. 6 and 7 can be shown tobe phase-shifted versions of the pulse-modulated waveform of FIG. 8,wherein the phase shift is proportional to the pulse width. Sincefrequency is the time-derivative of phase, this means if the pulse widthis varied in time, the waveforms of FIGS. 6 and 7 will befrequency-shifted by an amount proportional to the time-derivative ofthe pulse width variation. This is readily verified experimentally asthese frequency shifts are readily discernable to the human ear. Thismeans that the pulse modulation scheme of FIG. 8 gives a cleanersolution: modulations of pulse width will change the timbre but not thepitch, allowing the pitch and the pulse width to be varied as two fullyindependent, separately discernable information-carrying parameters.

FIG. 9 depicts generation of a pulse waveform generated from a thresholdcomparison of an adjustable threshold value with the amplitude of aperiodic sinusoidal waveform. Again the waveform may be an electricalquantity, non-electrical media quantity, or quantity associated withhigher-level signal attributes. A center-anchored of controllable width,PulseG_(a,b) (t) 902 is generated from the positive portions of aperiodic sine waveform 901 with similar comparator arrangements employedas were in the discussions of FIGS. 6 through 8. Here the periodic sinewaveform typically oscillates between a value of R 902.1 and a value of−R 902.2. The reference signal at a particular instant may be set at avalue equal to a proportion Rc 902.2 of this, 0≤c≤1. Presenting thesewaveforms to an appropriate comparator implementation whose outputvalues are A_(max)=A 902.3 and A_(min)=0 results in the depicted pulsePulseG_(a,b) (t) 902.

In FIG. 9, consider:

$a = {\frac{T}{2\pi}{{Arcsin}\lbrack c\rbrack}}$$b = {\frac{T}{2} - {\frac{T}{2\pi}{{Arcsin}\lbrack c\rbrack}}}$The pulse width is:

$\frac{T}{\pi}{{Arcsin}\lbrack c\rbrack}$and the duty cycle is:

$\frac{100\%}{\pi}{{Arcsin}\lbrack c\rbrack}$

As the reference signal Rc is raised (c approaches 1 or Rc approachesR), the region of the pulse wave with the amplitude of A_(max) getsnarrower, and the regions of the pulse wave with the amplitude 0 getwider. Similarly, as the reference signal Rc is lowered (c approaches0), the region of the pulse wave with the amplitude of A_(ma) getswider, and the regions of the pulse with the amplitude 0 get narrower.Also, because of the symmetry of the sine wave is similar to that in thetriangle wave of FIG. 8, it is readily proven that modulations of pulsewidth will change the timbre but not the pitch, allowing the pitch andthe pulse width to be varied as two fully independent, separatelydiscernable information-carrying parameters.

Should pulse-width modulation be used, it can be advantageous to usezero-DC pulse-width modulation as taught in U.S. patent application Ser.No. 12/144,480 entitled “Variable Pulse-Width Modulation with ZeroConstant DC Component in Each Period”, particularly if many such pulsewaveforms are summed together.

Sonification Sound-Field Audio Rendering

FIG. 10 shows “multi-channel sonification” using data-modulated soundtimbre classes set in a spatial metaphor stereo sound field. The outputsmay be stereo, four-speaker, or more complex, for example employing 2Dspeaker, 2D headphone audio, or 3D headphone audio so as to provide aricher spatial-metaphor sonification environment.

FIG. 11 shows an arrangement where dataset is provided to sonificationmappings controlled by interactive user interface. Sonification mappingsprovide information to sonification drivers, which in turn provideinformation to internal audio rendering and a MIDI driver.

Timbre Spaces and Sonic Design

The timbre of sounds is often mentioned in sonification, but aside fromleveraging varieties of timbral qualities of traditional musicalinstruments, the above work provides little in the way of systemic useof differentiated perception of the timbre of sounds as a tool forsonification in general (and, as discussed later, multi-channelsonification in particular). Notable exceptions to this include [1]based on the notion of Grey's “timbre space” abstraction [2]. Timbrespace has been further formalized in a way that characterizes it closeto that of a conventional linear vector space, including notions ofdistance [3-4] and its use as a control structure [5]. This worktypically again expresses varieties of timbral qualities in terms ofthose of traditional musical instruments, although [4] and (with regardsto timbre as an abstraction) [6] include and describe the synthesis ofsound timbre.

FIG. 12 shows an arrangement of a three-dimensional partitioned timbrespace. Here the timbre space has three independent perceptioncoordinates, each partitioned into two regions. The partitions allow theuser to sufficiently distinguish separate channels of simultaneouslyproduced sounds, even if the sounds time modulate somewhat within thepartition as suggested by FIG. 13. Alternatively, timbre spaces may have1, 2, 4 or more independent perception coordinates.

The features described thus far can readily be extended to clusters oftwo or more separately perceived sonification tones, each tone carryingits own set of information. FIG. 10 depicts an example multipleperceived parameterized-tone sonification rendering architecture whereineach sound source is specified by sound type (“class”) and whoseindividual timbre may be separately controlled according to variableparameters associated with that sound type. As an example, FIG. 13bshows a recasting of the timbre-space sonification trajectoryarrangement depicted in FIG. 13a , but with two separately perceivedparameterized-tone sources (i.e., N=2 for FIG. 10), and FIG. 13c shows acase with four separately perceived parameterized-tone sources (i.e.,N=4 for FIG. 10).

Other collections of audio signals also occupy well-separated partitionswithin an associated timbre space. A more sophisticated example of apartitioned timbre space technique also providing a partitioned spectralspace is the system and method of U.S. Pat. No. 6,849,795 entitled“Controllable Frequency-Reducing Cross-Product Chain.” The harmonicspectral partition of the multiple cross-product outputs do not overlap.

Through proper sonic design, each timbre space coordinate may supportseveral partition boundaries, as suggested in FIG. 12 and FIG. 14.Further, proper sonic design can produce timbre spaces with four or moreindependent perception coordinates.

Exemplary Pre-Sonification Operations

Attention is now directed to consideration of pre-sonificationoperations.

Data sonification (and data visualization) can be made far more powerfulif one or more mappings of data can be shifted, scaled, or warped withnonlinearities (such as logarithmic, exponential, or power-lawfunctions). Such functionality can be combined with indexing and sortingfunctions, as well as provisions for updating underlying datasets withnew measurements, trial synthesized data, and/or live sensor feeds.

FIG. 15 depicts an approach for mapping a data value lying within apre-defined range to a value within a pre-defined range for aparameterized data or presentation attribute. In most cases the inputdata range must be at least scaled and/or shifted so as to match thepre-defined range for a parameterized presentation attribute. In somecircumstances it may also be desirable to warp the data range with anonlinearity. A library of fixed or adjustable nonlinearities can beprovided such that the input and output of the nonlinearity both matchthe pre-defined range for a parameterized presentation attribute. Thewarping effect is provided with additional flexibility by allowingpre-scaling and/or pre-shifting prior to applying a selectednonlinearity and subjecting the outcome of the nonlinear warping topost-scaling and/or post-shifting operations in order to match theresulting range to the pre-defined range for a parameterizedpresentation attribute. An example of such arrangements is depicted inFIG. 15.

FIG. 16 depicts a more general view and organization of pre-sonificationoperations provided for by the invention. In this example, availablepre-sonification operations include:

-   -   Data indexing/reindexing, data sorting, data suppression, and        similar types of data operations;    -   Normalization, shifting (translation), and other types of linear        and affine transformations;    -   Linear filtering, convolution, linear prediction, and other        types of signal processing operations;    -   Warping, clipping, nonlinear transformations, nonlinear        prediction, and other nonlinear transformations.

Two or more of these functions may occur in various orders as may beadvantageous or required for an application and produce modified data.Aspects of these functions and/or order of operations may be controlledby a user interface or other source, including an automated dataformatting element or an analytic model. The invention further providesthat updates are provided to a native data set.

The invention also provides for other types of pre-sonificationoperations. Statistical operations and statistical processing functionscan be used as pre-sonification operations, and for linking to externalprograms to perform other types of pre-sonification operations. Externalprograms can be added to the collection of available pre-sonificationoperations.

FIG. 17 shows an arrangement wherein interactive user controls and/orother parameters are used to assign an index to a data set. Theresultant indexed data set is assigned to one or more parameters as maybe useful or required by an application. The resulting indexed parameterinformation is provided to a sound rendering operation resulting in asound (audio) output. In some embodiments provided for by the invention,the parameter assignment and/or sound rendering operations may becontrolled by interactive control or other parameters. This control maybe governed by a metaphor operation useful in the user interfaceoperation or user experience, as described later.

Sonification Time-Index Handling

In some situations, the data in a dataset to be sonified is definedagainst an intrinsic or inherent time-line. The sonification renderingin some cases may be performed at the natural time scale, at a speededor slowed time-scale, in real-time, or in artificially controlled time(as with a shuttle wheel control, animation loop, etc.). In othercircumstances the time-line may be artificially created from componentsof the data (for example, time may signify travel in distance, increasein temperature, etc.). Additional variations on these capabilitiesinclude the creation and use of artificial trajectories, such as thepath through the Belmont urban wetland slough depicted in FIG. 20. Herea user-defined geographic path trajectory can be traversed by a clockedor hand-manipulated travel rate, with location visually signified by acursor, arrow head, lengthening line, color change, or other means. Inresponse, an example multi-channel sonification may render three datavalues at any given position of the trajectory traversal (for exampledirect-measured and/or kriging-interpolated values of salinity,temperature, and turbidity) as three discernible timbre (or other)parameters of what is perceived as a single controllable tone source (asa simple example, employing (1) pulse width modulation depth, (2) pulsewidth modulation rate, and (3) distuned-dissonance created by additivesinewave difference effects). A resultant timbre trajectory of thesethree varying timbral parameters, with movement along the path of thistimbre trajectory tracking movement along the path of the user-definedgeographic trajectory of FIG. 20, could resemble the trajectory depictedin FIG. 13 a.

Rates of change of sound parameters can easily be even more of a concernin multi-channel and multiple-perceived-tone sonification. Due to theintertwined ˜20 Hz lowest perceived frequency and ˜50 msec timecorrelation window [64] of auditory perception, temporal variation oftimbre, pitch, and amplitude attributes at periods/rates notably lessthan 50 msec/20 Hz are perceived as a change in these attributes, whiletemporal variation of timbre, pitch, and amplitude attributes at ratesnotably more than 50 msec/20 Hz are perceived as quality of timbre ofthe tone as was illustrated in FIG. 4b . Thus, sonification time-indexhandling can provide more usable perception experiences if rates ofchange of tone attribute variations (and to some extent portions of avariation signal's instantaneous harmonic structure as well) are keptbelow the 50 msec/20 Hz rate. Alternatively, compensation can beprovided for faster rates of change; these can be quite sophisticated inform but are analogous to shifting the frequency of each harmonic of atone reproduced by a moving speaker so as to compensate for theinstantaneous Doppler shift resultant from the movement of the speaker.

As an example, the three dimensions of the timbre space may stillrepresent salinity, temperature, and turbidity and each of the (two orfour) separate sources represent different water depths or differingtransverse locations across the water surface. Although the samethree-parameter tone described in conjunction with FIG. 13a may be used(for example at different pitches or different spatial locations in astereo sound field), in general as more signal sources are added thesonic space becomes more cluttered. Additionally, if pitch is used asthe discerning attribute among the multiple instances of the same toneclass (type of sound), it is noted that in many circumstances modulationindexes must be adjusted with frequency so as to obtain the sameperceived effect.

The invention provides for each of these considerations, as well as farmore sophisticated and varied tone classes than the one described in theexamples of FIGS. 13a-13c , and in particular for effective mixtures ofsimultaneously sounding tone classes. For example, it may beadvantageous to superimpose one or more completely different soundingtone class(es) for representing other location-dependent attributes,such as (in an environmental GIS system as considered later) calculatedintegrated run-off drainage volume into the waterway, estimatedcross-sectional depth of the waterway, etc. It is noted that the lattertwo data examples (calculated integrated run-off drainage volume,estimated cross-sectional depth) may not be actual data in the datasetbut may be from another dataset or from run-time calculations made fromsimpler component data in the same or linked dataset.

Use of Metaphors

Accordingly, the invention additionally provides for the inclusion anduse of visual metaphors to simplify sonification setup and userinteraction for data exploration. As an example, FIG. 17 also depicts anarrangement wherein a selected metaphor is used to automaticallygenerate parameter assignments and graphics rendering operations. Theinvention provides for metaphors to control other aspects of thesonification and pre-sonification operations, and to base its operationson characteristics of a data set being visualized and/or sonified,previously visualized and/or sonified, and/or anticipated to bevisualized and/or sonified. Metaphors are selected and controlled byuser interaction, data values, or other means.

Use of Multidimensional User Interfaces

As mentioned above, the invention provides for the support, inclusion,and use of multidimensional user interface devices for providing extracontrol parameters, 3D-geometry control and metaphors, 6D-geometrycontrol and metaphors, etc. Such multidimensional user interface devicescan include a High-Definition Touchpad that taught in U.S. Pat. No.6,570,078, and U.S. patent application Ser. Nos. 11/761,978 and12/418,605, advanced computer mice taught in U.S. Pat. No. 7,557,797 andU.S. patent application Ser. No. 10/806,694, video cameras taught inU.S. Pat. No. 6,570,078, or other types of touch, control-based, orvisually operated user interfaces. The invention provides for theincorporation of and use of multidimensional user interface devices ininteracting with data visualization and/or data sonificationenvironments, either stand alone or in collaborative environments.

Use of Data Flow Paths to Implement Arbitrary Interconnection Topologies

The invention provides for the use of data flow paths to link arbitrarydata sources with arbitrary data destinations via arbitrary topologies.This allows the selection and/or fusion of data sources, theirinterconnection with selected signal processing, statistical processing,pre-sonification operations, and sonification parameters.

FIG. 18 depicts an topological interconnection of data flow pathslinking various elements that can be relevant in a data sonificationenvironment. Functions such as data reindexing, statistical processing,and signal processing can be used as data sources or as thepre-sonification functions. Similarly, numerical simulations, as may berendered by a high-performance or other computer, can serve as datasources. Certain pre-sonification functions, for example linearpredictors, may in an embodiment be regarded as a numerical simulation.

The invention provides for some or all of the data flow paths (such as agraphical diagrammic depiction of the arrangement of FIG. 18) to bespecified in any convenient way, for example graphically via aninteractive GUI or via a character-based language (interconnection,specification, and/or data-flow, etc.). A GUI can include the renderingof a graphic similar to that of FIG. 18, or can permit creation andcustomization of instances of functional blocks such as the onesdepicted in FIG. 18 from a library, menu, and/or graphical pallet. A GUIcan be used to create and link these customized instances of functionalblocks, via link-by-link “drawing,” with a data path topology such asthe ones depicted in FIG. 18.

Shared Data Sonification and Data Visualization Environments

FIG. 19 depicts an adaptation of the arrangement depicted in FIG. 18configured to selectively direct individually parameters to be renderedwithin a visualization, within a sonification, or within bothsimultaneously.

The example in the section to follow shows how this arrangement can beuseful in an application.

In situations where there is a natural or artificial timeline, theinvention provides for synchronization between data sonificationrendering and presentation and the data visualization rendering andpresentation.

Use of GIS and Data Visualizations as Interactive User interface forData Sonification

FIG. 20 depicts a data visualization rendering provided by a userinterface of a GIS system depicting an aerial or satellite map image fora studying surface water flow path through a complex mixed-use areacomprising overlay graphics such as a fixed or animated flow arrow. Thesystem can use data kriging to interpolate among one or more of storedmeasured data values, real-time incoming data feeds, and simulated dataproduced by calculations and/or numerical simulations of real worldphenomena.

FIG. 20 depicts an flow path that can be provided via a user interfacebuilt atop of and in terms of a GIS and/or data visualization. Thesystem can visually plot this data or use it to produce a sonification.Attention is directed to the flow path (curved arrow line) through avisual representation (here, a satellite image) of the environment areaunder study as shown in FIG. 20 as it would be on a user interfacedisplay.

The visual plot or sonification can render representations of one ormore data values associated with a selected point selected by a cursor acursor (shown as a small black box on the curved arrow line) on a flowpath (curved arrow line), or as a function of time as a cursor (shown asa small black box on the curved arrow line) moves along the flow path ata specified rate.

The system can visually display this data or use the data to produce asonification.

The sonification may render sounds according to a selected point on theflow path, or as a function of time as a cursor moves along the flowpath at a specified rate. For example, the system can produce atrajectory in sonification parameter (timbre) space such as thatdepicted in FIG. 12, wherein as a cursor moves along the path in FIG. 20the corresponding sonification rendered would simultaneously behave asprescribed by the trajectory in sonification parameter (timbre) spacedepicted in FIG. 12.

An embodiment of the invention can overlay visual plot items or portionsof data, geometrically position the display of items or portions ofdata, and/or use data to produce one or more sonification renderings.For example, a sonification environment may render sounds according to aselected point on the flow path, or as a function of time as a cursormoves along the surface water flow path at a specified rate.

Use of Auditory Perception Eigenfunctions

The invention provides for sonifications employing “auditory perceptioneigenfunctions” in the production of the data-manipulated sound. Astaught in that provisional patent application, these “auditoryperception eigenfunctions” are eigenfunctions (within a Hilbert space)for an operator equation defined by three of the most fundamentalempirical attributes of human hearing:

-   -   the approximate 20 Hz-20 KHz frequency range of auditory        perception [7];    -   the approximate 50 msec temporal-correlation window of auditory        perception (for example “time constant” in [8];    -   the approximate wide-range linearity (modulo post-summing        logarithmic amplitude perception, nonlinearity explanations of        beat frequencies, etc.) when several signals are superimposed        [7,8].

The audio perception eigenfunctions can be related to the integralequation whose eigenfunctions are the Prolate Spheroidal Wave Functions(“PSWFs,” also known more recently as “Slepian functions”) [9]. Theintegral equation for the audio eigenfunctions stems from a (typicallysmoothed) time-domain gate function and a (typically smoothed)frequency-domain bandpass function; in comparison, the integral equationwhose eigenfunctions are the PSWFs stems from an (abrupt) time-domaingate function and an (abrupt) frequency-domain lowpass function. As theauditory perception eigenfunctions are, by its very nature, defined bythe interplay of time limiting and band-pass phenomena, it is possiblethe Hilbert space model eigensystem may provide important newinformation regarding the boundaries of temporal variation and perceivedfrequency (for example as may occur in rapidly spoken languages, tonallanguages, vowel guide [10-12], “auditory roughness” [8], etc.), as wellas empirical formulations (such as critical band theory, phantomfundamental, pitch/loudness curves, etc.) [7,8].

While the invention has been described in detail with reference todisclosed embodiments, various modifications within the scope of theinvention will be apparent to those of ordinary skill in thistechnological field. It is to be appreciated that features describedwith respect to one embodiment typically can be applied to otherembodiments. Audio eigenfunctions are taught in the inventor's copendingU.S. patent application Ser. No. 12/849,013.

The invention can be embodied in other specific forms without departingfrom the spirit or essential characteristics thereof. The presentembodiments are therefore to be considered in all respects asillustrative and not restrictive, the scope of the invention beingindicated by the appended claims rather than by the foregoingdescription, and all changes which come within the meaning and range ofequivalency of the claims are therefore intended to be embraced therein.Therefore, the invention properly is to be construed with reference tothe claims.

REFERENCES

-   [1] Barrass, S.; “A perceptual framework for the auditory display of    scientific data,” ACM Transactions on Applied Perception, Vol. 2 No.    4, October 2005, pp. 389-402.-   [2] Grey, J. M., An exploration of musical timbre, Ph.D.    dissertation, Dept. of Music Report No. STAN-M-2, Stanford    University, CA, 1975.-   [3] Terasawa, H.; Slaney, M.; Berger, J., “Perceptual Distance in    Timbre Space,” ICADO5 Proceedings, 2005.-   [4] Drioli, C.; Polotti.; Delle Monache, S.; Rocchesso, D.;    Adiloglu, K.; Annies, R.; Obermayer, K.; “Auditory Representations    as Landmarks in the Sound Design Space,” Proc. SMC09, Porto, July    2009.-   [5] Wessel, D., “Timbre Space as a Musical Control Structure,”    Rapport Ircam 12/78, 1978    http://articles.ircam.fr/textes/Wesse178a/(visited Nov. 27, 2009).-   [6] Scaletti, C., “Sound synthesis algorithms for auditory data    representations,” Auditory Display: Sonification, Audification, and    Auditory Interfaces. G. Kramer, (Ed.), Santa Fe Institute Studies in    the Sciences of Complexity, Proc. Vol. XVIII. Addison-Wesley,    Reading, Mass.-   [7] Winckel, F., Music, Sound and Sensation: A Modern Exposition,    Dover Publications, 1967.-   [8] Zwicker, E.; Fastl, H., Psychoacoustics: Facts and Models,    Springer, 2006.-   [9] Slepian, D.; Pollak, H., “Prolate Spheroidal Wave Functions,    Fourier Analysis and Uncertainty—I,” The Bell Systems Technical    Journal, pp. 43-63, January 1960.-   [10] Rosenthall, S., Vowel/Glide Alternation in a Theory of    Constraint Interaction (Outstanding Dissertations in Linguistics),    Routledge, 1997.-   [11] Zhang, J., The Effects of Duration and Sonority on Contour Tone    Distribution: A Typological Survey and Formal Analysis (Outstanding    Dissertations in Linguistics), Routledge, 2002.-   [12] Rosner, B.; Pickering, J., Vowel Perception and Production    (Oxford Psychology Series), Oxford University Press, 1994.

What is claimed is:
 1. A data sonification system for representing aplurality of channels of numerical information via a plurality ofcorresponding discernable variations of timbre of at least one of aplurality of audio-frequency waveforms, the at least one of theplurality of the audio-frequency waveforms being perceivable by a useras comprising a plurality of audio tones having at least onecorresponding discernable timbre attribute, the data sonification systemcomprising: a plurality of audio-frequency waveform generators, each ofthe plurality of audio-frequency waveform generators generating anassociated one of the plurality of the audio-frequency waveforms,wherein each of the associated ones of the plurality of theaudio-frequency waveforms comprises a fixed audio-frequency and at leastone adjustable timbre control parameter, each of the at least one of theadjustable timbre control parameter having an associated adjustablevalue which can be discernibly varied within a timbre space occupied bythe plurality of the audio-frequency waveforms; and a mapping elementfor mapping aspects of multidimensional numerical data with the at leastone of the adjustable timbre control parameters of each of theassociated ones of the plurality of the audio-frequency waveforms,wherein for the associated ones of the plurality of the audio-frequencywaveforms the mapping element adjusts a value of the at least one of theadjustable timbre control parameters responsive to values of themultidimensional numerical data, wherein for the plurality of theaudio-frequency waveforms the timbre of the associated ones of theaudio-frequency waveform carries information responsive to values ofunderlying data for presentation to a user, and wherein the associatedones of the plurality of the audio-frequency waveforms and theirrespective adjustable timbres are arranged to provide a plurality ofdata-modulated sound timbre classes within the timbre space.
 2. The datasonification system of claim 1 wherein the at least one of the pluralityof the audio-frequency waveforms comprises a plurality of timbre controlparameters, each of the plurality of the timbre control parametersaffecting the timbre of the at least one of the plurality of theaudio-frequency waveforms; and wherein the mapping element adjusts thevalue of each of the plurality of the timbre control parametersresponsive to values of the multidimensional numerical data.
 3. The datasonification system of claim 1, wherein the sound timbre classes can bearranged to convey a sonic metaphor.
 4. The data sonification system ofclaim 1, wherein the values of the multidimensional numerical data aresequentially selected by the user from the multidimensional numericaldata, the values sequentially selected over an interval of time.
 5. Thedata sonification system of claim 1, wherein the values of themultidimensional numerical data are sequentially selected according to atrajectory selected through a subset of the multidimensional numericaldata.
 6. The data sonification system of claim 1, wherein the values ofthe multidimensional numerical data are sequentially selected responsiveto a position on a trajectory specified by the user through a subset ofthe multidimensional numerical data, the trajectory rendered in a datavisualization.
 7. The data sonification system of claim 1, wherein thevalues of the multidimensional numerical data are sequentially selectedresponsive to a position of a cursor within a rendered datavisualization.
 8. The data sonification system of claim 1, wherein themultidimensional numerical data is retrieved from storage.
 9. The datasonification system of claim 1, wherein the multidimensional numericaldata is provided by a live real-time data stream.
 10. The datasonification system of claim 1, wherein the multidimensional numericaldata is created from live sensor data.
 11. The data sonification systemof claim 1, wherein the mapping comprises pre-sonification operationsinvolving the imposing of indexing on selected data.
 12. The datasonification system of claim 1, wherein the mapping comprisespre-sonification operations involving data suppression.
 13. The datasonification system of claim 1, wherein the mapping comprisespre-sonification operations involving data value normalization.
 14. Thedata sonification system of claim 1, wherein the mapping comprisespre-sonification operations involving linear transformations.
 15. Thedata sonification system of claim 1, wherein the mapping comprisespre-sonification operations involving affine transformations.
 16. Thedata sonification system of claim 1, wherein the mapping comprisespre-sonification signal processing filtering operations on selecteddata.
 17. The data sonification system of claim 1, wherein the mappingcomprises pre-sonification operations involving nonlineartransformations.
 18. The data sonification system of claim 1, whereinthe mapping comprises pre-sonification operations involving linearpredictor operations.
 19. The data sonification system of claim 1,wherein the at least one timbre control parameter controls a width of apulse waveform.
 20. The data sonification system of claim 1, wherein atleast one of the plurality of the audio-frequency waveforms comprises aplurality of the adjustable timbre control parameters, each of theplurality of the adjustable timbre control parameter affecting thetimbre of the audio-frequency waveform; and wherein the mapping adjuststhe value of each of the plurality of the adjustable timbre controlparameters responsive to the values of the multidimensional numericaldata.